Testing is an important activity to be performed with any system that is being developed. One common practice is to design tests to be performed by testers or other QA personnel by creating test descriptions for the QA personnel to use. The designed tests can be stored and be used repeatedly, for example each time the system is updated or during regression tests.
Over time, there may be a considerable amount of tests relating to the system. Some features may change over time, potentially requiring adding, changing and removing some tests. In addition, as the tests may be designed by different people or even by the same person at different times, some of the tests may be redundant as they cover functionalities that are already covered by other features. On the other hand, although there may be a vast number of tests, some functionalities of the system may not be covered by any one of them.
As the test suite grows larger, managing the tests becomes an increased burden. It becomes harder and harder to identify which tests are missing, which have become redundant, whether tests may be consolidated, and so forth. This is in particularly true in large systems maintained by several different people whose identities change over time.
A model of the test-space, herein referred to as “model”, may be useful for assisting in test planning, coverage analysis, or other test related activities. The model may define tests that are within the test-space, based on any aspect of the tests, such as but not limited to inputs, scenarios, configurations, or the like. The model comprises a set of functional attributes, respective domains of values for each attribute, and potentially restrictions on the value combinations. Each attribute may relate to a different aspect of a test, such as, for example, operation to be performed, input to be provided, or the like. Each test is represented by a valuation to each of the attributes of the model that together define the functionality of the test. The model may be used to identify redundant tests, to identify missing tests or aspects of the system that are uncovered by the test suite, or the like.
In some cases, Combinatorial Test Design (CTD) may be used for test planning purposes based on the test suite. CTD may be based on a desired coverage goal of the model, which may be defined as a target level of interaction between the attributes (e.g., pair-wise interaction, 3 attributes interaction, or the like). Using CTD, additional tests may be defined to cover uncovered functionalities.
In some cases, test selection may be performed to reduce a size of the test suite without changing the covered test-space or while maintaining a same coverage level with respect to a coverage goal. The test selection may be performed based on the model of the test-space.